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STEPUP PROCEDURES FOR CONTROL OF GENERALIZATIONS 
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Stanford University 

Consider the multiple testing problem of testing null hypotheses 
Hi , . . . , Hs ■ A classical approach to dealing with the multiplicity prob- 
lem is to restrict attention to procedures that control the familywise 
error rate (FWER), the probability of even one false rejection. But 
if s is large, control of the FWER is so stringent that the ability of a 
procedure that controls the FWER to detect false null hypotheses is 
limited. It is therefore desirable to consider other measures of error 
control. This article considers two generalizations of the FWER. The 
first is the k-FWER, in which one is willing to tolerate k or more 
false rejections for some fixed fc > 1. The second is based on the false 
discovery proportion (FDP), defined to be the number of false rejec- 
tions divided by the total number of rejections (and defined to be 
if there are no rejections). Benjamini and Hochberg [J. Roy. Statist. 
Soc. Ser. B 57 (1995) 289-300] proposed control of the false discov- 
ery rate (FDR), by which they meant that, for fixed a, E{FDP) < a. 
Here, we consider control of the FDP in the sense that, for fixed 
7 and a, P{FDP > 7} < a. Beginning with any nondecreasing se- 
quence of constants and p-values for the individual tests, we derive 
stepup procedures that control each of these two measures of error 
control without imposing any assumptions on the dependence struc- 
ture of the p-values. We use our results to point out a few interesting 
connections with some closely related stepdown procedures. We then 
compare and contrast two .FiJP-controlling procedures obtained us- 
ing our results with the stepup procedure for control of the FDR of 
Benjamini and Yekutiefi [Ann. Statist. 29 (2001) 1165-1188]. 

1. Introduction. In this article we consider the problem of simultane- 
ously testing hypotheses Hi (i = 1, . . . , s). We shall assume that tests based 
on p-values pi,. . . ,ps are available for the individual hypotheses and that 
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the question of interest is how to combine these p-values into a simultane- 
ous testing procedure. In other words, each pi is a marginal p-value in the 
sense that it could be used for testing Hi] p- values for testing individual 
hypotheses are reviewed in Section 3.3 of [11]. 

A classical approach to handling the multiplicity problem is to restrict 
attention to procedures that control the familywise error rate (FWER), 
defined to be the probability of one or more false rejections. When evaluating 
a testing procedure, one must consider not only control of false rejections, 
but also the ability of the procedure to detect departures from the null 
hypothesis when they do occur. When the number of tests s is large, control 
of the FWER is so stringent that departures from the null hypothesis have 
little chance of being detected. As a result, alternative measures of error 
control have been considered, which control false rejections less severely, 
but in doing so are better able to detect false null hypotheses. 

Hommel and Hoffman [8] and Lehmann and Romano [10] considered con- 
trol of the k-FWER, the probability of rejecting at least k true null hy- 
potheses. Such an error rate with A; > 1 is appropriate when one is willing to 
tolerate one or more false rejections, provided the number of false rejections 
is controlled. Evidently, taking k = 1 reduces to the usual FWER. These 
authors derived both single step and stepdown methods that guarantee that 
the k-FWER is bounded above by a. 

Lehmann and Romano [10] also considered control of the false discovery 
proportion {FDP), defined as the total number of false rejections divided 
by the total number of rejections (and equal to if there are no rejections). 
Given a user specified value 7 € [0, 1], control of the FDP means we wish to 
ensure that P{FDP > 7} is bounded above by a. Setting 7 = reduces to the 
usual FWER. Lehmann and Romano [10] also provided stepdown procedures 
for control of the FDP that hold under either mild or no assumptions on the 
joint distribution of the p-values. Romano and Shaikh [12] improved upon 
these arguments to derive a stepdown procedure for control of the FDP 
that is also valid under no restrictions on the dependence structure of the 
p- values, but considerably more powerful than the method proposed in [10]. 
In this article, unlike either of these previous works, we consider stepup 
procedures. We derive stepup procedures that control the k-FWER and the 
FDP under no assumptions on the joint distribution of the p-values. 

A closely related type of error control that has received much attention 
since it was first proposed in [1] is control of the false discovery rate {FDR), 
which demands that E{FDP) is bounded above by a. This original paper 
imposed the very strong assumption that the p-values were independent, 
but Benjamini and Yekutieli [2] have since proposed a stepup method that 
is valid under no assumptions on the joint distribution of the p-values. It is 
of interest to compare control of the FDP with control of the FDR. Even 
though ensuring that the FDR is bounded does not prohibit the FDP from 
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varying, some obvious connections between methods that control the FDP 
in the sense that 

(1) P{FDP > 7} < a 

and methods the control its expected value, the FDR, can be made. Indeed, 
for any random variable X on [0, 1] , we have 

E{X) = E{X\X < -f)P{X < 7} + E{X\X > 7)P{X > 7} 

<7P{X <^} + P{X >j}, 

which leads to 

(2) SS^<p(x>,)<m, 

1-7 7 

with the last inequality just Markov's inequality. Applying this to X = FDP, 
we see that, if a method controls the FDR at level q, then it controls the FDP 
in the sense P{FDP > 7} < g/7. Conversely, if the FDP is controlled in the 
sense of (1), then the FDR is controlled at level 7(1 — a) + a. Therefore, in 
principle, a method that controls the FDP in the sense of (1) can be used to 
control the FDR and vice versa, as previously noted by van der Laan, Dudoit 
and Pollard [16]. We will compare methods for control of the FDP with the 
method for control of the FDR proposed by Benjamni and Yekutieli [2] in 
light of this observation. Note that setting a = 1/2 restricts the median of 
the FDP to be no greater than 7. 

A growing literature has proposed various procedures which control gen- 
eralized error rates. Genovese and Wasserman [4], for example, study asymp- 
totic procedures that control the FDP and the FDR in the framework of 
a random effects mixture model. These ideas are extended in [4]. Korn, 
Troendle, McShane and Simon [9] provide methods that control both the 
k-FWER and FDP; their results are limited to a multivariate permutation 
model. Their results are generalized in [13]. Alternative procedures for con- 
trol of the k-FWER and FDP are given in [16]. 

The paper is organized as follows. In Section 2 we describe our termi- 
nology and the class of stepup procedures. All of our methods assume that 
marginal p- values are available for testing each of the individual hypotheses, 
in the sense described in (3). Our methods are designed to hold under no de- 
pendence assumptions among the p- values, but do not attempt to estimate 
the dependence structure (as in van der Laan, Dudoit and Pollard [16] or Ro- 
mano and Wolf [13]). Hence, our main results are exact and nonasymptotic; 
however, if the individual p-values are only approximate (as they typically 
are when using asymptotic approximations or resampling methods), the er- 
ror control will hold approximately; see Remark 4.2. Control of the k-FWER 
and FDP are considered, respectively, in Sections 3 and 4. Our calculations 
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in these two sections shed some hght on the relationship between stepup 
and stepdown procedures as weh. In Section 5 we use the relationship (2) 
to compare methods for controlling the FDP with the method of Benjamini 
and Yekutieli [2] for controlling the FDR. Section 6 illustrates the method 
with two examples. Section 7 concludes. 

2. A class of stepup procedures. A formal description of our setup is 
as follows. Suppose data X is available from some model P € 0. A general 
hypothesis H can be viewed as a subset uj oi Vt. For testing Hi: P € Ui, 
i = 1, . . . , s, let I{P) denote the set of true null hypotheses when P is the 
true probability distribution; that is, i G I{P) if and only if P G coi. 

We assume that p- values pi, . . . ,ps are available for testing Hi, . . . , Hg. 
Specifically, we mean that pi must satisfy 

(3) P{Pi l^u} l^u for any u G (0, 1) and any P G Wj. 

Note that we do not require pi to be uniformly distributed on (0, 1) if Hi is 
true, in order to accommodate discrete situations. In deriving our results, 
we assume that (3) holds exactly, but we show in Remark 4.2 below that all 
of our results also extend to the case in which the p-values only satisfy (3) 
approximately. 

In general, a p-value pi will satisfy (3) if it is obtained from a nested set 
of rejection regions. In other words, suppose Si{a) is a rejection region for 
testing Hi] that is, 

(4) P{X G Si{a)} <a for all < a < 1, P G 

and Si{a) C Si{a') whenever a < a' . Then the p-value pi defined by pi = 
Pi{X) = inf {a : X G Si{Q)} satisfies (3). Such a construction applies to many 
parametric procedures and also some nonparametric procedures, such as 
those based on permutation or randomization tests; see (15.5) in [11]. 
In this article we consider the following class of stepup procedures. Let 

(5) ai<a2<- ■■ <as 

be a nondecreasing sequence of constants. Order the p-values as 

P(l) <P(2) <---<P(s), 

and let -ff(i), • • • , denote the corresponding null hypotheses. If < Og, 
then reject all null hypotheses; otherwise, reject hypotheses -ff(i), ... 
where r is the smallest index satisfying 

(6) > , • • • , P(r+1) > "r+l • 

If, for all r, p(^) > Or, then reject no hypotheses. That is, a stepup procedure 
begins with the least significant p- value and continues accepting hypotheses 
as long as their corresponding p-values are large. 
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We will compare these stepup procedures considered with certain step- 
down procedures. Given constants of the form (5), a stepdown procedure 
determines which null hypotheses to reject as follows. If > ai, then re- 
ject no null hypotheses; otherwise, reject hypotheses -ff(i), . . . ,-f^(r)) where r 
is the largest index satisfying 

(7) <ai,.-.,P{.r) <"r- 

That is, a stepdown procedure begins with the most significant p-value and 
continues rejecting hypotheses as long as their corresponding p-values are 
small. 

Remark 2.1. Consider a stepup and a stepdown procedure based on 
the same set of critical values (5). The stepup procedure will always reject 
at least as many hypotheses as the stepdown procedure. If both methods 
satisfy the given measure of error control, then the stepup procedure is more 
powerful than the corresponding stepdown procedure based on the same 
critical values in the sense that the stepup procedure will have a greater 
chance of detecting false null hypotheses. 

3. Control of the k-FWER. In this section we consider control of the 
k-FWER, defined formally as 

(8) Pjreject > k hypotheses Hi with i € I{P)}. 

Control of the k-FWER at level a requires that k-FWER < a for all P. 
We first establish a result that will aid in constructing stepup methods that 
control the k-FWER. 

Lemma 3.1. Consider testing s null hypotheses, with \I\ of them true. 
Let 

9(1) < • • • < m) 

denote the ordered values of the p-values corresponding to true hypotheses. 
Then, the stepup procedure based on constants ai < ■ ■ ■ < as satisfies 

(9) k-FWER<pl U {g(,)<a,_|,|+,}l. 

U-<i<|/| J 

Proof. Assume that |/| > k, for otherwise there is nothing to prove. 
Let < • • • < P(^s) denote the ordered values of the p-values. For 1 < j < s, 
let Aj denote the event in which exactly j hypotheses are rejected by the 
stepup procedure; that is, 

= {P{s) >as,... ,P{j+i) > <^j^P{j) < "j}- 
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Denote by T the event in which at least k true hypotheses are rejected. 
Consider the event AsHT. Note that AsHT Q {^(5) < aJflT C {q^^j^-^ < as}. 
Likewise, note that ^s-i n T C {q^^j^_i^ < Us-i} if |/| — 1 > A; and C {g^^.^ < 
q;<j_i} if |/| — 1 < k. In general, we have that 

^ -\{q(k)<aj}, if j <s-\I\+k. 

Thus, the k-FWER is bounded above by the probability of the event 

[j Aj'^TqI U %.)<a,}|u| U %+|,|_,)<a,}| 

k<j<s '^k<j<s-\I\+k ) U-|/|+fc<i<s ) 

^ U {% + |/|-.) <aj} ^ U 

s-\I\+k<j<s k<j<\I\ 

where the second inclusion follows from the fact that {q(^k) ^ Oj} C {g^^^ < 
a<j_[/[+fc} for J < s — |/| + k. The asserted claim now follows. □ 

Given a sequence of constants ai < • • • < a^i we will now use Lemma 3.1 
to construct a stepup procedure that controls the k-FWER. To this end, 
define 



(10) Si = Si{k,s,\I\) = \I\^^^ + \I\ 

k<j<\i\ 
and let 



F 



(11) Di = Di{k,s)= max 5i(A;, s, |/|). 

fc<|/|<s 

Theorem 3.1. Let ai < ■ ■ ■ < as be given. For testing Hi'.P^ uji, i = 
1, . . . ,s, suppose Pi satisfies (3). Consider the stepup procedure with critical 
values a'j^ = aai/Di{k, s) , where Di{k,s) is defined by (11). 

(i) Then k-FWER < a. 

(ii) Moreover, for any stepup procedure with critical values of the form 
6ii = aai/D' for some constant D' that satisfies k-FWER < a, we have for 
each i that a[>ai. 

Before proceeding with the proof of Theorem 3.1, we recall the following 
lemma from [10], which generalizes an earlier result from [7]. 

Lemma 3.2. Suppose pi, ... ,pt are p-values in the sense that P{pi<u} <u 
for all i and u in (0, 1). Let their ordered values be < • • • < pi^t) ■ For some 
m<t, let 

= /3o < /3i < /32 < • • • < /3m < 1. 
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(i) Then 

P{{Pil) < Pi} U {^(2) < /32} U • • • U {p(^) < Pm}} 

(12) 

m 

<tY^{(3i-Pi-i)/i. 
1=1 

(ii) As long as the right-hand side of (12) is < 1, the bound is sharp in 
the sense that there exists a joint distribution for the p-values for which the 
inequality is an equality. 

Proof of Theorem 3.1. (i) Combining Lemmas 3.1 and 3.2, we have 
that 



k-FWER<p\ U %)<aU|,|+^.}| 



^ ^ j2 "'^-i^i+j ^'s^\i\+j~i 

^ k<i<\i\ ^ 

a 



-Si{k, s, \I\) < a. 



Di{k,s) 

(ii) Consider the following joint distribution of p-values. Denote by \I\* 
the value of |/| maximizing 51(^,5,1/1). Let the p-values of the s — \I\* false 
hypotheses be identically equal to (or just < a'l) and let the p-values of 
the \I\* true hypotheses be constructed according to part (ii) of Lemma 3.2 
so that 

P\ U {Q{j)<^s^\i\'+i}\ = T^Si{k,s,\I\*) = ^a, 
lfc<j<|/|* ) 



where the second equality uses the fact that Oj = aai/D' . For such a joint 
distribution of p-values, the event of rejecting > k true hypotheses is equiv- 
alent to rejecting > s — \I\* -\- k hypotheses in total. So, 

k-FWER = pl U {g(^.)<d,_|,|.+^.}| = :^a. 

U<j<|/|* J 

Thus, to ensure control of the k-FWER, it must be the case that Di < D' . 
It follows that, for each i, a',>ai. □ 



Theorem 3.1(ii) shows that it is not possible to increase all of the critical 
values by any amount without violating control of the k-FWER. In this 
sense, part (ii) of the theorem represents a sort of weak optimality result. 
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Table 1 

Values of Di(k, s) for k-FWER control with ai given by (13) and (19) 



fc = 1 



k = 2 



k = 3 



s 



(13) 



(19) 



(13) 



(19) 



(13) 



(19) 



10 
25 
50 
100 
250 



2.11 
2.13 
2.13 
2.13 
2.13 



3.92 
7.99 
14.52 
27.32 
65.25 



2.03 
2.16 
2.16 
2.16 
2.16 



2.57 
4.72 
8.10 
14.63 
33.77 



1.90 
2.15 
2.17 
2.17 
2.17 



2.10 
3.60 
5.91 
10.33 
23.22 



500 


2.13 


128.08 


2.16 


65.34 


2.17 


44.36 


1000 


2.13 


253.41 


2.16 


128.17 


2.17 


86.35 


2000 


2.13 


503.75 


2.16 


253.51 


2.17 


170.01 


5000 


2.13 


1254.20 


2.16 


628.96 


2.17 


420.46 



Hommel and Hoffman [8] and Lehmann and Romano [10] propose using 
constants proportional to 



as part of a stepdown procedure to control the k-FWER and showed that 
such a procedure using critical values aai controls the k-FWER at level 
a under no assumptions on the joint distribution of the p-values. We can 
apply Theorem 3.1 to this choice of ai to construct a step up procedure that 
also controls the k-FWER under no restrictions on the joint distribution 
of the p- values. Table 1 displays for several different values of k and s the 
normalizing constant Di{k,s) of Theorem 3.1. Table 1 shows that the con- 
stants must be approximately halved to ensure control of the k-FWER. For 
example, in the case s = 1000 and k = 3, the optimizing value of \I\ is 39, 
yielding L>i(3, 1000) = 2.1707. 

For control of the FWER, Hochberg [5] proposed using the stepup pro- 
cedure with critical values given by (13) with k = 1. These same constants 
were used by Holm [6] to control the FWER, but as part of a stepdown 
procedure. Hochberg argued that his procedure controls the FWER assum- 
ing that the p-values are independent. Sarkar and Chang [15] have shown 
that Hochberg's procedure also controls the FWER for certain forms of pos- 
itively dependent p- values. So, it follows from Remark 2.1 that under such 
assumptions on the joint distribution of the p-values Hochberg's procedure 
is more powerful than the one proposed by Holm. Holm's procedure, how- 
ever, controls the FWER under no assumptions on the joint distribution of 
the p-values, whereas our results show that this is not true of Hochberg's 



(13) 
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procedure. However, we show that by dividing the constants by Di{l,s), 
control of the FWER is restored. 

Remark 3.1. The notion of control that we consider demands that 
k-FWER < a for all P. This is sometimes referred to as strong control of k- 
FWER in order to distinguish it from a weaker (and not particularly useful 
for multiple testing) notion of control known as weak control of the k-FWER, 
where it is only required that the k-FWER < a for all P satisfying |/| = 
|I(P)| = s, that is, when all hypotheses are true. The distinction between 
weak and strong control generalizes in an obvious way to measures of error 
control other than the k-FWER. It is interesting to note that to guarantee 
even weak control of the k-FWER, the constants aoi, where Oi is defined 
by (13), must be approximately halved (at least for large s). To see this, 
first note that when |/| = s, the k-FWER is equivalent to the probability of 
rejecting > k hypotheses altogether; that is, 

(14) pI U %)<Qa,}|. 

i.k<j<s ) 

Using Lemma 3.2, we know there exists a joint distribution of the p-values 
for which (14) is equal to 

a[l + s ^I^^] 

V k<j<s ) 

(15) =a(l + fc y — ^— — -) 

= ah+fc y + y (, + A:-i)i(i + l)) 

\ k<i<s ^ ' k<i<B ^ J \ < J / 

Ck , .t-^ i — k \ 
2-- + k y — . 

But, it is easy to see that, as s ^ oo, we have that (15) 2a. It follows 
that, at least for large values of s, the constants aoj must be approximately 
halved to ensure weak control of the k-FWER. In fact, the expression (15) 
is strictly larger than the limiting value 2a, and so the constants must be 
divided by something slightly greater than 2. In order to guarantee strong 
control, the constants must be divided by something that is only slightly 
larger than 2. In the case k = 1, this value is 2.1314. 

Remark 3.2. More generally, suppose \I\ is not neccesarily = s and 
denote the ordered values of the true p- values by < • • • < ^([/|)- Then, 
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following the argument given in the proof of Theorem 3.1(ii), we have that 
(16) k-FWER = pl y %)<aa,_,,|+,}|. 

Again, Lemma 3.2 asserts that there exists a joint distribution of true 
p- values for which (16) is equal to 



(17) 

Note that 




k 



so we may use the analysis of Remark 3.1 with the role of s replaced by |I| 
to conclude that (17) is equal to 

(18) af2-A)+of^l°^l^l 



I\J V 1^1 

If |/| is large, then it is sufficient to halve the constants aai to control the 
k-FWER approximately. The expression (18) implies further that if we index 
both k and |/| by the number of hypotheses s and allow s — > oo, then the 
stepup procedure with critical values aai/2 provides strong control of the 
k-FWER, provided that /c '°^|'^ — > 0. Division by 2 can be thought of as the 
price to pay for using a stepup versus stepdown procedure (based on the 
same set of critical values). It is perhaps surprising that the value of 2 is 
independent of the choice of k. 

Finner and Roters [3] compared stepup and stepdown procedures for con- 
trol of the EWER assuming that the p-values are exchangeable. Under the 
setup of their paper, their results suggest that stepup procedures are more 
powerful than stepdown procedures because one can use very nearly the 
same critical values for both procedures to control the EWER. However, in 
our comparisons, we assume nothing about the joint distribution of p- values 
and find that in such a setting the stepup procedure requires smaller crit- 
ical values (by roughly a half) to provide control of the FWER, and more 
generally of the k-FWER. 

We may also apply Theorem 3.1 to the sequence of constants given by 

(19) ai = -. 

s 

The normalizing constant Di{k,s) for this choice of is also displayed in 
Table 1. In light of part (ii) of Theorem 3.1, we should not expect either of 
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the sequences of critical values generated by applying Theorem 3.1 to (13) 
and (19) to be uniformly larger (and thus unambiguously more powerful) 
than the other. In order to illustrate this fact, we plot the two sequences 
of constants for the case in which k = 2, s = 100 and a = 0.05. Panel (a) of 
Figure 1 displays the constants based on (13), whereas panel (b) displays the 
constants based on (19). Panel (c) depicts the ratio of the constants in panel 
(a) with the constants in panel (b). The dashed horizontal line in panel (c) is 
of height 1, allowing us to see graphically when the constants from panel (a) 
are greater than the constants from panel (b) and vice versa. We find that, 
for high and low values of i, the constants based on (13) are larger than the 
constants based on (19). For intermediate values of i, where the constants 
based on (13) are smaller than the constants based on (19), the differences 
between the constants are quite small in absolute terms, whereas, for other 
values of i, the differences between the constants are fairly substantial. This 
suggests that the procedure based on (13) may be preferable to the one 
based on (19). 

4. Control of the FDP. The number of false rejections that one is will- 
ing to tolerate will often increase with the number of hypotheses rejected. 
This leads to consideration of not the number of false rejections (sometimes 
called false discoveries), but rather the proportion of false discoveries. For- 
mally, let the false discovery proportion (FDP) be defined by 



FDP is therefore the proportion of rejected hypotheses that are rejected 
erroneously. When none of the hypotheses is rejected, both numerator and 
denominator of that proportion are 0; since, in particular, there are no false 
rejections, the FDP is then defined to be 0. We now establish a general result 
that will aid us in constructing stepup procedures that control the FDP in 
the sense of (1). In what follows, we will sometimes use m(j) as shorthand 
for ['yj\ + 1, where [x\ is the greatest integer < x, and the notation x\/ y 
in place of max{x,y}. 

Lemma 4.1. Consider testing s null hypotheses, with \I\ of them true. 
Let ^(1) < ■ • ■ < ^(|/|) denote the ordered p-values corresponding to true hy- 
potheses. Then the stepup procedure based on constants ai < ■ ■ ■ < as satis- 
fies 



(20) 




if the denominator is > 



if there are no rejections. 



P{FDP > 7} 



(21) 





Fig. 1. Stepup constants for k-FWER control with k — 2, s — 100 and a = 0.05. 



Proof. Let Aj be the event in which exactly j hypotheses are rejected 
by the stepup procedure; that is, 



= {P(s) > tts, • • • ,P{j+i) > Oij,P{j) < aj}. 
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Let Tj be the event in which at least m{j) true hypotheses are rejected. 
Then 

Likewise, for the event Ag-i nTs_i, we have that 

^ < "s-i}, if 1^1 - 1 > m{s - 1), 

C {(?(|/|) < a^-i}, if |/| = m(s - 1), 

= 0, otherwise. 

It follows that 

As^i n Ts^i C {p(s„i) < as_i} n Ts_i 

C {g((|/|_i)vm{s-i)) < as-i}, if |/| > m(s - 1), 
= 0, otherwise. 

Following a similar line of reasoning, we have in general that 

C {g((|j|_j)v.m(s-j)) < "s-j}, if \I\ > ?7i(s - j), 
= 0, otherwise. 

Thus 

{FDP > 7} C IJ {^((|7|„j)v™(s_j)) < Os-j} 

0<j<s~l,\I\>m{s-~j) 

= U {9{fcVm(s-I/I+fc)) < Ols-\I\+k}, 

\I\-s+l<k<\I\,\I\>m{s~\I\+k) 

from which the asserted claim follows. □ 

Given a sequence of constants ai < • • • < as, we will now use Lemma 4.1 
to construct a stepup procedure that satisfies (1). To this end, define 

S2 = S2h,S,\I\) 

(22) 

= |/|ai + |/| )^ fcVmfs-lJl+fc) 

\I\-s+l<k<\I\,\I\>mis-\I\+k) ^Vm(S |i|+/Cj 

and let 

(23) D2 = D2{^,s)= max 52(7,s,|/|). 

1<|/|<S 
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Theorem 4.1. Let ai < ■ ■ ■ < as be given. For testing Hi'.P uji, i = 
1,.. .,s, suppose Pi satisfies (3). Consider the stepup procedure with critical 
values a'- = aui / D2{'y , s) , where 02(^,3) is defined by (23). 

(i) Then P{FDP > 7} < a; that is, (1) is satisfied. 

(ii) Moreover, for any stepup procedure with critical values of the form 
Ui = aai/D' for some constant D' that satisfies (1), we have for each i that 
a'l>ai. 

Proof, (i) Combining Lemmas 4.1 and 3.2, we have that 

P{FDP > 7} < ^1 U %Vn^(.-|/|+fc)) < 

^I\-s+l<k<\I\,\I\>m{s-\I\+k) J 



<\iwi + \i\ E 



^"s-\I\+k "s-|/|+fc-l 



|/|_,+i<fc<|/|,|/|>,„(,_|/|+fe) ^Vm(s \I\ + k) 
01 

32(7,3, \I\) < a. 



D2{7,s 

(ii) Consider the following joint distribution of p-values. Denote by |/|* 
the value of |/| maximizing 5*2(7, s, |/|). Let the distribution of the p- values 
corresponding to the \I\* true hypotheses be constructed according to part 
(ii) of Lemma 3.2 so that 

P\ U {Q{kVm{s-\I\*+k)) < Ois-\I\*+k} \ 

I |/|*-s+l<fc<|/|*,|/|*>m{s-|7|*+fc) J 

"en ITI*\ -^2 
= ^S2{k,s,\I\ ) = jyO:, 

where the second equality uses the fact that = aai/D' . We will now 
construct the joint distribution of the p- values corresponding to the s — \I\* 
false hypotheses conditional on the values of the true p- values so that FDP > 
7 whenever (21) occurs. For the time being, suppose that |/|* is such that 
|/|* > m{s). Thus, (21) can be written more simply as 

P{FDP >7}<P\ U {q{kVmis-\I\*+k)) < J 1 ■ 

I |/|*-s+l<fc<|/|* ) 

Define k* to be the smallest index k > such that k > m{s — \I\* + k). 
Consider the event 

(24) IJ {q(kWm{s-\I\*+k)) < «s-|/|*+fc} = U {^(fc) - 0^s-|/|*+fc}- 
|/|*>fc>fc* |/|*>fc>fc* 
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Whenever the event (24) occurs, let all false p-values be identically equal to 
0. By assumption, k > m{s — \I\* + k) and > 0, so note that whenever this 
event occurs, we have that FDP > 7. 

Now suppose that the event (24) does not occur. Note that this rules out 
the possibility of any event of the form 

{Q{kVm{s-\I\*+k)) < 

for k < k* and k V m{s — |/|* + A;) = A;* . So, let k** be the largest k < k* such 
that k V m(s — |/|* + k) = k* — 1 and consider the event 

(25) {q{k*-i) < as-\i\*+k**}- 

Whenever the event (25) occurs but (24) does not, let s - + A;** - /c* + 1 of 
the false p-values be identically equal to and let the remaining k* — k** — 1 
false p- values fall between ag_\i\*_^_i^** and ag_\j\*_^_i.* . Again, by construction, 
whenever (24) does not occur but (25) does occur, we have FDP > 7. 

We may continue arguing along these lines by replacing the role of k* 
with k** to construct a joint distribution of false p-values conditional on the 
true p-values such that, whenever (21) occurs, we have that FDP > 7. But 
we have assumed so far that \I\* >m{s). To generalize the argument to the 
case in which |/|* < ■m{s), note that the event (21) is always of the form 

U {q{k)<ai{k)} 

l<k<\I\* 

for some strictly increasing sequence of positive integers /(I) < • • • < /(|/|*). 
Thus, the smallest /(|/|*) can be is |/|*. Let {s-\I\*)-{s-l{\I\*)) = 1{\I\*)- 
\I\* > of the false p- values be identically equal to 1. Since these hypotheses 
will always be accepted by the stepup procedure, we can restrict attention 
to the situation in which there are s — /(|/|*) + |/|* hypotheses altogether, 
s — ^(|/|*) of which are false. But for this situation, our assumption on the 
number of true hypotheses holds, so we may use the construction above to 
determine the distribution of remaining false p- values. 
So, for such a joint distribution of p-values, we have that 

P{FDP > 7} = IJ {q{kVm{s-\I\*+k)) < as-|7|*+fc} 

Thus, to ensure control the FDP, it must be the case that D2 < D' . It follows 
that, for each i, a'- > Oi. □ 

Lehmann and Romano [10] develop a stepdown procedure that controls 
the FDP in the sense of (1) by reasoning as follows. Denote by F the number 
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of false rejections. At step i, having rejected i — 1 hypotheses, we want to 
guarantee F/i < 7, that is, F < [72] . So, if A; = [jil + 1, then F >k should 
have probability no greater than a; that is, we must control the number of 
false rejections to be < k. This leads them to consider using the stepdown 
constants (13) for control of the k-FWER with this particular choice of k 
(which now depends on i). That is, 

(26) ai — 



s + [7iJ +1-1 

Lehmann and Romano [10] provide two results that show that the step- 
down procedure with critical values aai with this choice of satisfies (1). 
Unfortunately, some assumption on the joint dependence structure of the 
p-values is required. However, they show that if one considers a stepdown 
procedure with critical values aa^ /C|^^sJ+i) where 

^ 1 

1=1 

then the FDP is controlled in the sense of (1) without any assumptions on 
the dependence structure of the p- values. 

Romano and Shaikh [12] show that this procedure is more conservative 
than necessary to control the FDP. Specifically, they show that the stepdown 
procedure with critical values obtained by replacing C^^j,j_|_i with a smaller 
quantity 1)3(7, ^) ^-Iso provides control of the FDP without any assumptions 
on the joint distribution of the p- values. This change leads to a considerable 
improvement, resulting in critical values typically 50 percent larger. 

We can apply Theorem 4.1 to ai defined by (26) to construct a stepup 
procedure that controls the FDP in the sense of (1). The normalizing con- 
stant D2{'j, s) is computed for several different values of 7 and s in Table 2. 
The column labeled "-02, (26)" refers to the value of D2 when the constants 
(26) are used. For the purposes of comparison, we also display 1)3(7,5). For 
large values of s, the normalizing constant D2{^,s) is strictly smaller than 
C[^^5j_i_i, but it is always larger than 1)3(7,5). Thus, it follows from Remark 
2.1 that, for large values of s, the stepup procedure is more powerful than 
the stepdown procedure proposed by Lehmann and Romano [10], whereas a 
clear ranking of the procedure relative to the stepdown procedure proposed 
by Romano and Shaikh [12] is not possible. 

As before with the k-FWER, we may also apply Theorem 4.1 to the 
sequence of constants defined by (19). The normalizing constant 1)2(7, s) 
for this choice of is also displayed in Table 2. Again, the optimality result 
stated in part (ii) of Theorem 4.1 suggests that we should not expect either 
of the sequences of critical values generated by applying Theorem 4.1 to (26) 
and (19) to be uniformly larger (and thus unambiguously more powerful) 
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Table 2 



Stepup constants for FDP control with cti given by (26) and (19) 



7 = 0.05 



7 = 0.1 



s 



£»2, (26) D2, (19) £>3, (26) D2, (26) D2, (19) D3, (26) 



10 
25 
50 
100 
250 
500 
1000 
2000 
5000 



2.11 
2.40 
2.70 
2.96 
3.41 
3.80 
4.24 
4.72 
5.39 



3.91 
7.99 
14.12 
20.32 
31.04 
40.33 
50.40 
61.05 
75.80 



1.00 
1.43 
1.50 
1.73 
2.12 
2.50 
2.92 
3.38 
4.044 



2.11 
2.68 
2.99 
3.37 
3.93 
4.39 
4.89 
5.41 
6.14 



3.91 
7.78 
10.96 
15.09 
21.21 
26.33 
31.75 
37.37 
45.06 



1.00 
1.50 
1.75 
2.04 
2.52 
2.95 
3.42 
3.92 
4.62 



than the other. We plot the two sequences of constants for the special case 
in which 7 = 0.1, s = 100 and a = 0.05. Panel (a) of Figure 2 displays the 
constants based on (26), whereas panel (b) displays the constants based 
on (19). Panel (c) depicts the ratio of the constants in panel (a) with the 
constants in panel (b). The dashed horizontal line in panel (c) is of height 1, 
allowing us to see graphically when the constants from panel (a) are greater 
than the constants from panel (b) and vice versa. We find that, for high and 
low values of i, the constants based on (26) are larger than the constants 
based on (19). But, as with the comparison of the k-FWER controlling 
procedures, we find that the differences are comparatively small when the 
ones based on (26) are smaller than the constants based on (19) and fairly 
large otherwise. Thus, we believe the procedure based on (26) is likely to be 
preferred to the one based on (19). 

Remark 4.1. Benjamini and Yekutieli [2] propose using the constants 
aoi/Cs for Oj given in (19) as part of a stepup procedure to control the 
FDR and show that such a procedure controls the FDR for all possible 
distributions of p-values. Since FDR = FWER when |/| = s, we have that 
these critical values also control the FWER when |/| = s. But, the results 
in Table 1 show that these constants do not control the FWER in general 
since Di(l,s) > Cg for this choice of a^. More surprising, however, is that 
this observation continues to be true even if one assumes that the p-values 
are independent. To see this, consider the case in which s is even and |/| = 
s/2 + 1. Suppose that all of the false p- values are identically equal to 0, and 
the true p- values, whose ordered values are denoted by < • • • < are 



each ~ f7(0, 1). Thus, 





r a ] fa Y'^+' 

Thus, the step up procedure with critical values aoi/Cg does not control the 
FWER, even under independence. 
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Remark 4.2. In many situations the true individual p- values do not 
satisfy (3) exactly. However, suppose the p-values Pj-"-* are now indexed by 
n (typically the sample size), and assume 

(27) lim < li} < ti for any u,P GtOi. 

n — ^oo 

For example, if the values are determined by an asymptotic method such as 

(n) 

the bootstrap, then it is typically the case that p] converges in distribution 
to the uniform distribution on (0, 1) if Hi is true. If we use a stepup procedure 
that controls the FDP for nominal values of a and 7 whenever the p- values 
satisfy (3) exactly, then we can claim limiting control if we use a stepup 
procedure based on p- values which only satisfy (27). Specifically, we claim 
that asymptotic control holds; that is, 

(28) lim sup P{ FLIP (p(")) > 7} < a, 

n—foo 

where the event {FDP(p^"''') > 7} that the FDP is not controlled now shows 
the dependence on n in that we are applying the procedure to the ap- 
proximate vector of p- values p^^^ = {pi^\ . . . ,pi"'^). To see why, let g^") = 
{q^\ . . . jq'^^^) denote the p- values corresponding to the true hypotheses, 

with ordered values ^^^j* < • • • < ^(|/|)- Then, by Lemma 4.1, 

p{FZ)P(pW) > 7} < ^|u41 ^ 

for some nondecreasing Pk- We can write the right-hand side as Pjt^^") G 
C}, where C is a closed set. (Note that the event that the FDP is not 
controlled, viewed as a set in s-dimensional space, is not a closed set; there 
is no contradiction since the set C corresponds to a larger set where the FDP 
is not controlled.) But, q^"'^ is a tight sequence in |/|-dimensional Euclidean 
space (since it is supported on a fixed compact set, the |/|-fold product of 
[0, 1]). So, taking any subsequence {rij}, there exists a further subsequence 
{nj; } along which converges in distribution to a random vector q = 
{qi, ■ ■ ■ ,q\i\) (which could depend on the subsubsequence) . Moreover, the 
assumption (27) implies (3) holds for each qi. Let the ordered values of q be 
denoted < • • • < ^(|/|)- By the Portmanteau theorem, it follows that 

limsupP{gK) eC}< P{q G C}=p\\Jq^k) < Pk] ■ 

I k J 

But, the right-hand side here is bounded above by a by Theorem 4.1. Since 
the bound a holds along any subsequence, the result is proved. A similar 
remark holds for control of the k-FWER when using p-values that only 
satisfy (27) instead of (3). 
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Table 3 

Minimum and maximum values of ratios of Benjamini-Yekutieli constants and constants 
based on (26) and (19) when both are used to control the FDP 



s 




7 = 0.05 






7 = 0.1 




min (26) 


max (26) 


(19) 


min (26) 


max (26) 


(19) 


10 


9.25 


27.76 


14.96 


4.63 


13.88 


7.48 


25 


4.71 


31.86 


9.55 


2.33 


14.26 


4.91 


50 


2.75 


33.40 


6.37 


1.99 


15.05 


4.10 


100 


2.25 


35.02 


5.11 


1.86 


15.41 


3.44 


250 


2.03 


35.80 


3.93 


1.75 


15.53 


2.88 


500 


1.95 


35.72 


3.368 


1.68 


15.46 


2.58 


1000 


1.88 


35.30 


2.97 


1.62 


15.30 


2.36 


2000 


1.81 


34.67 


2.68 


1.58 


15.10 


2.19 


5000 


1.73 


33.74 


2.40 


1.52 


14.82 


2.02 



5. Comparisons of FDP and FDR control. In the previous section we 
have put forward two stepup procedures [one based on (26) and another 
based on (19)] that control the FDP in the sense of (1) under no assumptions 
on the dependence structure of the p- values. In this section we will use 
the crude inequalities given in (2) to compare these two FZ)P-controlling 
procedures with the FZ)i?-controlling stepup procedure of Benjamini and 
Yekutieh [2]. 

From the second inequality in (2), control of the FDR at level 7a implies 
control of the FDP in the sense of (1). Since the Benjamini and Yekutieli 
stepup procedure with constants ai/{sCs) controls the FDR at level a, the 
constants given by 

(29) oii = — 

SL^s 

control the FDP under no assumptions on the joint distribution of the p- 
values. We will first compare these critical values with the critical values of 
the form aai / D2{'y , s) derived by applying Theorem 4.1 to ai defined by 
(26). Note that the ratio of the critical values aai / D2{'y , s) to a'^ is only a 
function of 7 and s. Table 3 displays for several different values of 7 and 
s the minimum and maximum values of this ratio. For all values of 7 and 
s in the table, the minimum value of the ratio > 1. In fact, the value of 
aai/D2{'y,s) is often at least twice as large as the corresponding value of 
a^. The procedure based on the constants (26) is therefore unambiguously 
more powerful than the procedure based on the constants (29). By examining 
the maximum value of the ratio, we see that the value of aai/ D2{'j, s) may 
be more than 15 times as large as the corresponding value of a'i- 

We may replace the critical values based on (26) with those based on (19) 
and perform the same comparison. In this case, the ratio of aai/ 02(^,3) to 
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ol^ is simply Cs/{'^D2{'^.,s)) and does not depend on i. Table 3 also displays 
the value of this ratio for several values of 7 and s. We find that the critical 
values aai 1^2(^1-, s) are always at least twice as large as the critical values a^; 
thus, as before, the procedure based on the constants (19) is more powerful 
than the procedure based on the constants (29). 

It is also possible to utilize the FZ)P-controlling constants to control the 
FDR, by application of (2). Now using (29) results in larger critical values 
than those resulting from application of Theorem 4.1. Detailed numerical 
comparisons are available from the authors. These results, though based 
on the crude inequalities in (2), suggest that it is perhaps worthwhile to 
consider the sort of control desired when choosing critical values. Indeed, 
the previous comparisons are somewhat unfair in that the FDi?-controlling 
procedures were not designed to control the FDP, and vice versa. 

However, we consider one final comparison in which the FZ)P-controlling 
constants are utilized to control the median of the FDP at level 7 by setting 
a = 1/2. We may compare these critical values with the Benjamini-Yekutieli 
critical values given by a'l = ji/sCs, which control the FDR at level 7. First, 
we consider the constants based on (26). Table 4 displays the minimum and 
maximum values of the ratio of these critical values to the critical values 
a'- for several different values of 7 and s. We find that, for moderate values 
of s, the critical values based on (26) are uniformly larger than the critical 
values a'- , but, for large values of s, the critical values a" are larger for 
some values of i. To examine whether these differences are of any practical 
significance, we plot in Figure 3 the two sequences of constants for the case 
in which s = 1000 and 7 = 0.1. Panel (a) displays the critical values based 
on (26), whereas panel (b) displays the critical values a'- . Panel (c) displays 
the ratio of the constants in panel (a) with the constants in panel (b). The 
dashed horizontal line in panel (c) is of height 1. It is clear that, except 
for some small values of i, the constants of panel (a) are often dramatically 
larger than the constants of panel (b). More importantly, at such values 
of i, the differences between the two sequences of critical values are quite 
small. Thus, for most practical purposes, the stepup procedure based on the 
constants in panel (a) seems preferable to the one based on the constants in 
panel (b). 

We now consider the same comparison with the critical values based on 
(26) replaced by the critical values based on (19). For this choice of the 
value of the ratio of the constants derived from Theorem 4.1 to the constants 
a'- no longer depends on i. Table 4 displays the values of this ratio for several 
values of 7 and s. Here we find that the critical values based on (19) used to 
control the median of the FDP are always uniformly larger, and therefore 
more powerful, than the FZ)i?-controlling critical values a'- , though, for large 
values of s, the two sequences of critical values are nearly indistinguishable. 
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Table 4 

Minimum and maximum values of ratios of Benjamini-Yekutieh constants when used to 
control the FDR and constants based on (26) and (19) when used to control the median 

of the FDP 



s 




7 = 0.05 






7 = 0.1 




min (26) 


max (26) 


(19) 


min (26) 


max (26) 


(19) 


10 


4.63 


13.88 


7.48 


2.31 


6.94 


3.74 


25 


2.36 


15.93 


4.78 


1.17 


7.13 


2.45 


50 


1.37 


16.70 


3.18 


1.00 


7.52 


2.05 


100 


1.12 


17.51 


2.55 


0.93 


7.71 


1.72 


250 


1.02 


17.90 


1.97 


0.88 


7.77 


1.44 


500 


0.98 


17.86 


1.68 


0.84 


7.73 


1.29 


1000 


0.94 


17.65 


1.49 


0.81 


7.65 


1.18 


2000 


0.91 


17.34 


1.34 


0.79 


7.55 


1.09 


5000 


0.87 


16.87 


1.20 


0.76 


7.411 


1.01 



6. Empirical applications. 

Example 6.1 (Benjamini-Hochberg application). We revisit the study 
of treatments for myocardial infarction analyzed in Benjamini and Hochberg 
[1], Section 3.2. For the 15 reported p- values, the Benjamini-Hochberg FDR 
controlling procedure at level 0.05 rejects four hypotheses. But, this proce- 
dure does not work for all possible joint distributions of p- values. The more 
generally applicable Benjamini- Yekutieli procedure rejects only three hy- 
potheses. In contrast, our procedure for controlling the median of the FDP 
at level 0.05 still rejects four hypotheses. 

Example 6.2 (Comparing strategies to a benchmark). The problem con- 
sidered is to determine which, if any, of several financial strategies outper- 
forms a given benchmark. The data set is similar to that in Romano and 
Wolf [14]. We consider all s = 210 hedge funds in the Center for Interna- 
tional Securities and Derivatives Markets (CISDM) database that have a 
complete return history from 01/1994 to 12/2003. All returns are net of 
management and incentive fees. The benchmark is the risk free rate of re- 
turn. Performance is measured monthly, so each fund has a return history 
of 120 values. It is well known that returns of hedge funds exhibit nontrivial 
serial correlations and the distribution of (Studentized) differences in log re- 
turns between a particular strategy and a benchmark must take into account 
such dependence. Individual or marginal p-values were calculated according 
to the Studentized circular block bootstrap, as reviewed in [14]. 

For k-FWER control at level a = 0.1 our stepup procedure rejects 10, 20 
or 24 hypotheses according to k = 1, k = 5 or k = 10. For control of the FDR, 
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Fig. 3. Stepup constants with median of FDP < 7 and FDR < 7 for s — 1000 and 
7 = 0.1. 

the Benjamini-Yekutieli procedure rejects 0, 16 or 23 according to whether 
7 = 0.01, 7 = 0.05 or 7 = 0.1, respectively. For control of the median of the 
FDP, our procedure rejects 20, 22 or 24 hypotheses according to the same 
values of 7 = 0.01, 7 = 0.05 or 7 = 0.1. 

7. Conclusion. In this article we have described stepup procedures for 
testing multiple hypotheses that control either the k-FWER or the FDP 
without any restrictions on the joint distribution of the p- values. For each of 
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these two measures of error control, we have also shown that the procedures 
constructed using our results satisfy a sort of weak optimality in that the 
critical values cannot all be made larger without violating the measure of 
error control. Our results have also revealed that control of the k-FWER or 
FDP using a stepup procedure assuming nothing about the joint distribu- 
tion of p- values requires smaller critical values than a step down procedure 
satisfying the same measure of error control. Finally, we have compared two 
Fl?P-controlling procedures obtained using our results with the stepup pro- 
cedure for control of the FDR of Benjamini and Yekutieli [2], which is also 
valid under no assumptions on the joint distribution of the p-values. These 
comparisons suggest that it is indeed important to consider the sort of error 
control desired when constructing multiple testing procedures. 

Acknowledgment. Thanks to Michael Wolf for computation of the p- values 
in Example 6.2. 

REFERENCES 

[1] Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A 
practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 
57 289-300. MR1325392 

[2] Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in 
multiple testing under dependence. Ann. Statist. 29 1165-1188. MR1869245 

[3] Finner, H. and Roters, M. (1998). Asymptotic comparison of step-down and step- 
up multiple test procedures based on exchangeable test statistics. Ann. Statist. 
26 505-524. MR1626043 

[4] Genovese, C. and Wasserman, L. (2004). A stochastic process approach to false 
discovery control. Ann. Statist. 32 1035-1061. MR2065197 

[5] Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of signifi- 
cance. Biometrika 75 800-802. MR0995126 

[6] Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scand. J. 
Statist. 6 65-70. MR0538597 

[7] HOMMEL, G. (1983). Tests of the overall hypothesis for arbitrary dependence struc- 
tures. Biometrical J. 25 423-430. MR0735888 

[8] HOMMEL, G. and Hoffman, T. (1987). Controlled uncertainty. In Multiple Hypoth- 
esis Testing (P. Bauer, C. Hommel and E. Sonnemann, eds.) 154-161. Springer, 
Heidelberg. 

[9] Korn, E., Troendle, ,J., McShane, L. and Simon, R. (2004). Controlling the 
number of false discoveries: Application to high-dimensional genomic data. J. 
Statist. Plann. Inference 124 379-398. MR2080371 

[10] Lehmann, E. L. and Romano, J. P. (2005). Generalizations of the familywise error 
rate. Ann. Statist. 33 1138-1154. MR2195631 

[11] Lehmann, E. L. and Romano, J. P. (2005). Testing Statistical Hypotheses, 3rd ed. 
Springer, New York. MR2135927 

[12] Romano, J. P. and Shaikh, A. M. (2004). On control of the false discovery propor- 
tion. Technical Report No. 2004-31, Dept. Statistics, Stanford Univ. 

[13] Romano, J. P. and Wolf, M. (2005a). Control of generalized error rates in multiple 
testing. Technical Report 2005-2012. Dept. Statistics, Stanford Univ. 



STEPUP CONTROL OF GENERALIZED ERROR RATES 



25 



[14] Romano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data 

snooping. Econometrica 73 1237-1282. MR2149247 
[15] Sarkar, S. and Chang, C. (1997). The Simes method for multiple hypothesis testing 

with positively dependent test statistics. J. Amer. Statist. Assoc. 92 1601-1608. 

MR1615269 

[16] van der Laan, M., Dudoit, S. and Pollard, K. (2004). Augmentation procedures 
for control of the generalized family-wise error rate and tail probabilities for 
the proportion of false positives. Stat. Appl. Genet. Mol. Biol. 3 Article 15. 



MR2101464 



Department of Statistics 
Stanford University 
Stanford, California 94305-4065 
USA 

E-MAIL: romano@stanford.edu 



Department of Economics 
Stanford University 
Stanford, California 94305-6072 
USA 

E-MAIL: ashaikh@stanford.edu 



